Index Structures for Information Filtering Under the Vector Space Model

Under the Vector Space Model Tak W. Yan and Hector Garcia-Molina Department of Computer Science Stanford University Stanford, CA 94305 Abstract With the ever increasing volumes of electronic information generation, users of information systems are facing an information overload. It is desirable to support information ltering as a complement to traditional retrieval mechanism. The number of users, and thus pro les (representing users' long-term interests), handled by an information ltering system is potentially huge, and the system has to process a constant stream of incoming information in a timely fashion. The e ciency of the ltering process is thus an important issue. In this paper, we study what data structures and algorithms can be used to e ciently perform large-scale information ltering under the vector space model, a retrieval model established as being e ective. We apply the idea of the standard inverted index to index user pro les. We devise an alternative to the standard inverted index, in which we, instead of indexing every term in a pro le, select only the signi cant ones to index. We evaluate their performance and show that the indexing methods require orders of magnitude fewer I/Os to process a document than when no index is used. We also show that the proposed alternative performs better in terms of I/O and CPU processing time in many cases.